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WHERE THE RUBBER MEETS THE ROAD 


Operations and Quality Control in School Accountability Systems’ 


EXECUTIVE SUMMARY 


Since the passage of the Every Student Succeeds Act (ESSA) in December 2015, state education 
agencies (SEAs) have invested time, effort, and resources into designing their accountability 
systems so that they reflect the state’s vision and priorities and are in compliance with federal 
statute. States, especially SEA staff members, should be applauded and congratulated for getting 
to this point — the development and approval of a state ESSA plan is an important milestone and 
key to supporting a variety of educational initiatives. However, it is only the start. To successfully 
meet the vision, priorities, and goals of the state accountability system requires an effective 
implementation plan. Much like in the design phase, implementation requires thoughtful 
consideration and intentional planning so that the system's annual outcomes reflect its intended 
design. And, as with the design phase, most states have a short amount of time to get their 
systems up and running. ESSA requires states to begin identifying schools for comprehensive 
support and improvement (CSI) by the 2018-2019 academic year’. This means that operational 
infrastructures, such as data and reporting systems, processes, business rules, and validation 
procedures, need to be in place soon. 


In support of the state’s validity argument for accountability, its implementation plan should 
reflect the design of the accountability system with fidelity. This means that the scores and ratings 
for schools and districts are correctly computed, schools and districts in need of support are 
appropriately identified, and that the claims made about schools and districts by the system are 
accurate. This paper outlines considerations for meeting these goals. Because each state has 
unique priorities and requirements for its accountability system, the paper is not a step-by-step 
“how-to” manual or specification for operational implementation and quality control. Instead, 

it describes a framework that states can use to guide the development of their accountability 
implementation plan and put guardrails in place to validate the various outcomes of the 
accountability systems. 


Figure 1 is a visual representation of a framework for a state’s accountability implementation 
workflow. The workflow includes three main stages: input, process, and output. Each stage 
includes components such as data files, data systems, business rules, reported data, and 
reporting system. 


1 This is the first of a three-paper set of resources presented at CCSSO's State Plan Implementation for ESSA 

in April 2018. The other resources include http://www.ccsso.org/resource-library/establishing-performance- 
standards-school-accountability-systems and http://www.ccsso.org/resource-library/accountability-identification- 
only-beginning. 

2 For more information on the timeline of the implementation of accountability systems and the identification 
of schools for support and improvement please review the United States Department of Education’s (USED) Dear 
Colleague letter on the 2017-2018 Transition here. 


Input Process Output 


Data Files Business Rules Reported Data 
Data Systems Reporting System 


Figure 1. An Accountability Implementation Workflow Framework 


The full paper explicates the framework by first stating the high-level objective for each component 
and then elaborating on the specifics by asking guiding questions about the following five W's 
from the state’s organizational structure and processes. 


¢ What: the key tasks or elements in this stage; 


¢ Who: the people, department or organization responsible for the tasks or elements in 
this stage. Note that this can include people outside of an SEA, which would require 
additional collaboration and coordination efforts; 


e When: the timeline or due dates for key tasks or elements in this stage; 


e Where: the sources of data, documentation and other relevant materials and resources 
for this stage; and, 


e Why: the rationale for key decisions related to this stage. 


The objectives and guiding questions in the framework are important for practitioners to consider 
not only in the initial planning and implementation of the state’s accountability system, but also for 
the ongoing monitoring, evaluation, and continuous improvement of the system. 


A key characteristic that underlies the objectives for all stages of the operational implementation 
workflow is the commitment to quality. Threats to the quality of an accountability system include 
errors in assessment or accountability data, misspecification or misunderstanding of business rules, 
and lack of stability in the data or reporting systems. Without a comprehensive and actionable 
plan for quality control, the state’s accountability system is vulnerable to errors or outages. Given 
the broad impact, high profile, and politically-charged nature of assessment and accountability in 
K-12 education, a few quality issues could lead to mistrust or fuel opposition, undermining even the 
most carefully designed and technically sound accountability systems. 


The final part of this paper offers recommendations for best practices to help mitigate or minimize 
the threats to quality in accountability implementation. The recommended practices are rooted in 
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tried and true quality control procedures or processes from implementing operational assessment 


programs and include: 


Issues tracking logs to record defects or unexpected activities and outcomes so that 
follow-up can occur to mitigate any potential risks; 


Specifications that document in detail the steps for all tasks in the implementation plan; 


A replication process in which multiple people are assigned to independently carry out 
the steps described in the specifications and verify that they yield the same results; 


Test cases that represent common or typical sets of condition as well as atypical, 
extreme or even out of bound conditions to determine whether accountability system is 
operating as intended; and, 


A reasonableness review process that takes a more macro view of the accountability 
results by considering the meaning and implications of the outcomes and looking for 
patterns or trends that are unusual. 


The full paper expands on each of these practices by providing guidelines and considerations for 


implementation in an accountability setting. 


Having a solid implementation plan and sound quality control processes are instrumental to 


meeting the vision, priorities, and goals of the state’s accountability system. Many states already 


have implementation plans and processes in place for their accountability systems. States are 


encouraged to share their resources and lessons learned with one another to leverage their 


experiences and to promote a culture of collaboration as each state enters the next phase of 


implementing its thoughtfully-designed accountability system. 


BACKGROUND 


Since the passage of the Every Student Succeeds Act (ESSA) in December 2015, state education 
agencies (SEAs) have invested much time, effort, and resources into designing their accountability 
systems so that they reflect the state’s vision and priorities and are in compliance with federal 
statute. Over the course of the past year, each state has submitted its consolidated ESSA plan to 
the United States Department of Education (ED) and has either completed or will be completing 
the approval process (see https://www2.ed.gov/admins/lead/account/stateplan17/index.htm)). 


States, especially SEA staff members, should be applauded and congratulated for getting to this 
point — the development and approval of a state ESSA plan is an important milestone and key to 
supporting a variety of educational initiatives. However, it is only the start. To successfully meet the 
vision, priorities, and goals of the state accountability system requires an effective implementation 
plan. Much like in the design phase, implementation requires thoughtful consideration and 
planning so that the system's annual outcomes reflect its intended design. And as with the design 
phase, most states have a short amount of time to get their systems up and running. ESSA requires 
states to begin identifying schools for comprehensive support and improvement (CSI) by the 
2018-2019 academic year (see https://www2.ed.gov/policy/elsec/leg/essa/essatranistiondcl11817. 
pdf). This means that operational infrastructures, such as data and reporting systems, processes, 
business rules and validation procedures, need to be in place soon. 


The purpose of this brief is to outline considerations for the operationalization and quality control 
of a state accountability system. Because each state has unique priorities and requirements 

for its accountability system, the brief is not intended to be a step-by-step “how-to” manual 

or specification for operational implementation and quality control. Instead, it describes a 
framework that states can use to guide the development of their accountability implementation 
plan and put guardrails in place to validate the various outcomes of the accountability system. 
Please note that while the context for this brief is the implementation of each state’s ESSA-based 
accountability system, the principles and practices apply broadly to other types of assessment 
and accountability systems. 


GOALS OF OPERATIONS AND QUALITY CONTROL 


Validity is the ultimate goal of any operational implementation plan and quality control process. 
The essential idea of validity is straightforward: it simply means that an instrument measures what it 


is purported to measure. 


If we view a state’s accountability system as an instrument that measures the degree to which schools 
and districts meet the state’s educational goals and priorities, then validity implies the following: 


1) The design of the accountability system is based on a theory of action that reflects 
the state’s goals and priorities and includes technically-sound components such as 
indicators, ratings, performance levels, and rules for identification. 
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2) The implementation of the system reflects the design of the accountability system with 
integrity. This means that the scores and ratings for schools and districts are correctly 
computed, schools and districts in need of support are appropriately identified, and 
that the claims made about schools and districts by the system are accurate. 


3) The ongoing monitoring and evaluation of the system shows that it is helping the state 
meet its goals and priorities. This includes a method of evaluating the system's utility 
and impact on districts and schools and a process for continuous improvement. 


The focus of an operational implementation plan and quality control process is the second aspect 
of a valid accountability system. In the sections to follow, there is a framework for structuring 

the operational workflow of a state’s accountability system implementation and suggested best 
practices for quality control. 


OPERATIONAL WORKFLOW 


The figure below is a visual representation of a framework for a state’s accountability 


implementation workflow. 


Input Process Output 


Data Files Business Rules 


Data Systems 


Reported Data 
Reporting System 


Figure 1. An Accountability Implementation Workflow Framework 


This workflow includes three main stages: input, process, and output. Each stage includes components 
such as data files, data systems, business rules, reported data and reporting system. One systematic 
approach that may be helpful to an SEA as it examines and elaborates on the specifics in each stage is 
to consider the following five W's from its organizational structure and processes. 


e¢ What: the key tasks or elements in this stage; 


e Who: the people, department or organization responsible for the tasks or elements at 
this stage. Note that this can include people outside of an SEA, which would require 
additional collaboration and coordination efforts; 


e When: the timeline or due dates for key tasks or elements in this stage; 


e Where: the sources of data, documentation and other relevant materials and resources 
for this stage; and, 


e Why: the rationale for key decisions related to this stage. 


The five W's within each stage are important to consider not only in the initial planning and 
annual implementation of the state’s accountability system, but also for the ongoing monitoring, 
evaluation, and continuous improvement of the system. 


In the proceeding description of each stage, a high-level objective for each component in the 
stage is provided. This is then followed by guiding questions, organized using the five W's 
approach, that an SEA can consider in its implementation plan to help meet the objective. Example 
solutions or responses to the guiding questions are given, as applicable. 


Input 


The input stage includes the various types of data that are used for accountability purposes (data 
files) and the systems used to capture, store and manage the data (data systems). The high-level 
objectives and associated guiding questions for the input stage include: 


Data Files 


Objective The data files are complete and correct. 


Guiding Questions 


What e¢ What data elements are needed to compute the measures and indicators, and 
implement the business rules in the accountability system? 


o  Student-level files that include the current-year demographic information 
(gender, race/ethnicity, free and reduced lunch status, special education 
status, English learner status, etc.) and statuses (full-year enrollment, 
attendance rates, etc.) 


Student-level files that include the scores and/or performance levels on the 
state's content-area assessments (general and alternate), English language 


proficiency assessments, and postsecondary readiness measures or assessments 
(such as the SAT, ACT, AP. IB, etc.) 


School-level files that include information about school characteristics (Title 
| status, grades served, etc.) and performance (graduation rates, chronic 
absenteeism rates, data from school climate surveys, CTE courses, etc.) 


What documentation or resources are available to help interpret or decode the values in 
the data files? 


o File layouts 


o Coding schemes for variables 


o Data dictionaries 
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Data Files 


When 


Data Systems 


The data systems are working and managed securely. 


Guiding Questions 
What 


Who is responsible for providing the various data files? What are the consequences for 
districts or schools that fail to meet the deadlines or submit the required data? 


Who is responsible for validating that the data files provided include all the required 
fields? (See the next section for more information about quality control practices.) 


Who is responsible for evaluating the integrity of the data files? 


Who should be contacted if the data files are missing, incomplete or include errors? 


When are the various types of data expected to be available? 

Student demographics 

School characteristics 

State content-area assessment scores 

Graduation rates 

Chronic absenteeism data 

SAT and ACT scores 

School climate survey results 
When are initial data files (for testing/dry runs) expected to be available? 
When are the final data files (for production runs) expected to be available? 


When should all the data files be ready for the processing stage? 


Where can additional documentation about the data files be found? 


Where should reports that document the completeness and integrity of the data files be 
archived? 


If there are multiple potential sources for a data file (or specific fields in the file), why 
was the source of data chosen for accountability implementation? 


Why are certain data files not available until given dates each year? 


Why do data files need to be ready each year by a given date? (What is the driver for 
when data processing needs to begin each year?) 


On what system does each type of the data file or element reside? 
What are different ways that each data system may be accessed? 
o Web-based vs. local client or application 

o Different devices (desktops, laptops, tablets, mobile devices) 
o Different platforms (Windows, iOS, Android, Linux) 

What are the security protocols for accessing each data system? 


What back-up procedures or redundancies are in place for each system in case of 
extended outages? 


If data files or elements needs to be shared across systems, what procedures are in 
place to maintain interoperability of the systems? 


Data Systems 


Who has ownership over each data system? That is, who is responsible for 
maintaining and updating the system, controlling user access, and providing 
technical support? 


o = State vs. district vs. school 
o Internal (SEA) vs. external (test vendor or technology contractor) 


Who should have access to each data system and what types of access (administrator, 
read-only) should they have? 


SEA personnel, such as the chief information officer (CIO), technology, data, and/ 
or program area staff. 


District schoo! administrators and testing coordinators 
Test vendor 


Technology contractor 


By when do districts and schools need to enter or verify their data in the systems? 


By when should the data systems be ready each year for accountability 
implementation? 


If routine system maintenance tasks, such as software updates or hardware updates, 
are needed, when should they be conducted to minimize disruptions? 


Where can documentation about each data system be found? 


Where are the servers for the data systems physically located? 


Why were the current set of data systems (and owners) chosen for accountability 
implementations? 


Why do the different levels of users have access to the data system? 
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Process 


The process stage includes the steps, computations, and logic (business rules) involved in taking 
the data from the input stage and generating the results needed for the output stage. The high- 
level objectives and associated guiding questions for the processing stage include: 


Business Rules 


Objective The business rules reflect the design of the accountability system accurately and 
poompienelsle 


What are the rules for cleaning and processing the data for accountability 
implementation? 


Excluding student or school records 

Handling duplicate student or school records 
Merging student or school records across data files 
Reconciling inconsistencies within or across data files 


What are the formulas and logic for computing each measure and indicator in the 
accountability system? 


Minimum n-size 

Weighting scheme 

All students and by student groups 
Aggregating data across years 


Exception handling (such as different grade configurations in schools, missing data, 
etc.) 


What are the rules for classifying and identifying schools for support? 


Indicator ratings and/or performance levels 
Summative school ratings and/or performance levels 


Decision rules for comprehensive support and improvement (CSI), targeted support 
and improvement (TSI) and additional targeted support and improvement (ATS]) 
Who is responsible for generating the specifications that describe in detail all of the 
business rules in the state’s accountability system? 
Who is responsible for implementing the business rules? 


Who is the independent replicator(s)? 


Who needs to validate and sign off on the results before they are passed on to the 
reporting systems (output stage)? (See the next section for more information about 
quality control practices.) 


When should testing or dry runs of the business rules be conducted? 


By when do the official (production) runs of the business rules need to be completed 
and the results signed off on? 


In which years are CSI, TSI, and ASTI identification required? 


Business Rules 


Where is the most recent version of the specifications stored? What is the process for 
version control? 


Where should key files (such as analysis outputs and summary reports) and documents 
(such as the issues tracking log) for the processing stage be archived? 


Where can support for the formulas and models used in the accountability system be 
found? 


o Technical documentation 
o Research support 


o Peer-reviewed publications 


What is the rationale for the various business rules in the accountability system? 
o Data cleaning 
o Indicator computations 


o School classification and identification 


What is the rationale behind the criteria for matching across independent data analysts? 


On what basis did the reviewer/evaluator sign off on the results in each year? Did the 
reviewer/evaluator note any concerns about the results? 
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The output stage includes the communication (reported data) and delivery (reporting systems) 


of accountability system results to its users. The high-level objectives and associated guiding 


questions for the output stage include: 


Objective 


Guiding Questions 


What 


Who 


The reported data are complete, accurate and correctly interpreted. 


What accountability outcomes should be reported? 

o Indicator scores and classifications 

o Summative rating and classification 

o Identification status (e.g., CSI, TSI, or ATS!) 

o Normative information (e.g., percentile ranks, comparison to a norm group, etc.) 
o Longitudinal trends 

o Additional key measures or metrics 

At what levels should the accountability outcomes be reported? 

o District vs. school vs. building vs. teacher 

o All students vs. student groups 

o By gender, ethnicity, socio-economic status (SES), or special populations 
o Across grade levels vs. by grade level 

o Across content areas vs. by content area 

o Current year vs. previous year(s) 


What documentation or resources are available to help users interpret and appropriately 
use the accountability results? 


o On-site visits and training session by SEA staff 
0 Interpretive guides 

o Informational webinars 

o How-to videos 


o Social media 


Who is responsible for validating the reported accountability outcomes? (See the next 
section for more information about quality control practices.) 


Who is responsible for approving (or locking down) the accountability results for 
reporting? 


Who is responsible for communicating the accountability results? 

o Summarizing key outcomes and overall trends 

o Explaining the interpretations and implications of the accountability results 
o Highlighting appropriate and inappropriate uses of the accountability results 


Who should be contacted if there are questions, concerns, or errors found in the 
reported accountability results? Is there an appeals process? 


Reported Data 


When are accountability outcomes from the processing stage expected to be ready 
each year? 


By when should accountability results be locked down and shared with users? 


By when do documentation and resources about the accountability results need to 
available? 


Where can accountability results be obtained? 

o SEA website 

o District or school administrative office (physical copies) 
o Outside publications 


Where can documentation and resources about the accountability results be found? 


Why were specific accountability results chosen for reporting? 
Why are specific accountability results not reported? 


Why do accountability results need to be reported each year by a given date? (What is 
the driver for annual accountability reporting?) 


Reporting Systems 


The reporting systems are working and accessible by all users. 


Guiding Questions 


What What system is used for accountability reporting purposes? 
What are different ways that reports can be accessed by users? 
o Web-based vs. local client or application 
o Different devices (desktops, laptops, tablets, mobile devices) 
o Different platforms (Windows, iOS, Android, Linux) 


What type of user interface (Ul) does the reporting system have? How user friendly and 
accessible is the reporting system UI%? 


Static vs. customizable reports 
Dashboards 

Interactive graphics 

Zoom-in and drill-down features 
Multiple languages 
Accessibility features 


What back-up procedures or redundancies are in place for the reporting system in case 
of extended outages? 


3 Agood reference for best practices in developing state report cards can be found here: http://www.ccsso.org/ 
resource-library/communicating-performance-best-practices-resource-developing-state-report-cards 
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Reporting Systems 


Who owns the reporting system? That is, who is responsible for maintaining and 
updating the system, controlling user access, and providing technical support? 


o SEA (internal) 

o External vendor or contractor 

Who should have different levels of access to the reporting system? 
o Administrative vs. read-only user access 


o Public vs. secure pages 


By when does the reporting system need to be tested each year? 

o Alpha vs. beta testing 

o Automated vs. user testing 

By when should the reporting system be ready each year for public access? 


If routine system maintenance tasks, such as software updates or hardware updates, 
are needed, when should they be performed to minimize disruptions? 


Where can the different types of documentation (e.g., administrator manual, user 
guide, Q&A document, etc.) and support (e.g., help desk, technical support page, etc.) 
about the reporting system be obtained? 


Where are the servers for the reporting system physically located? 
Why was the current reporting system (and owner) chosen for accountability 
implementation? 


Why do the different types of users have their respective levels of access to the 
reporting system? 


BEST PRACTICES FOR QUALITY CONTROL 


A key characteristic that underlies the objectives for all stages of the operational implementation 
workflow is the commitment to quality. Threats to the quality of an accountability system include 
errors in assessment or accountability data, misspecification or misunderstanding of business rules, 
and lack of stability in the data or reporting systems. Without a comprehensive and actionable plan 
for quality control, the state’s accountability system is vulnerable to errors or outages. And given 
the broad impact, high profile, and politically-charged nature of assessment and accountability in 
K-12 education, a few quality issues could lead to mistrust or fuel opposition, undermining even the 
most carefully designed and technically sound accountability systems. 


Well-defined and executed quality control procedures are part and parcel to the annual design, 
administration, scoring, and reporting processes of most assessment programs. Many of the quality 
control practices for assessment programs are applicable to accountability systems. Here are some 
best practices for quality control that an SEA may consider incorporating into its accountability 
implementation plan. 


Issues Tracking Logs. These usually take the form of a shared spreadsheet or database in which any 
issues related to the annual implementation of the accountability system are recorded and tracked. 
Each entry in the log should include fields such as the date of the issue (date), a short narrative of 
the issue (description), person or group assigned to the issue (responsible party), tasks or processes 
impacted by the issue (risks), updated status of the issue (status), and a summary or comment about 
the mitigation approach (resolution). Keeping track of issues, large or small, is not only a critical 
practice for a successful rollout of the accountability system in the current year, but also helps build 
a knowledge base for accountability implementation in the years to come. One important factor 
that contributes to the utility of the tracking log is ownership. That is, a person or group should be 
designated as the owner of the log, responsible for maintaining, updating, and following up with the 
people or groups assigned to each issue. The log can be used as the primary point of reference for 
regular touchpoints, such as daily calls or emails during peak processing or implementation periods. 
The logs can often be used as documentation for auditing purposes as well. 


Specifications. The importance of sufficiently detailed documentation for any type of 
implementation plan cannot be overstated. Specifications are “how-to” documents that include 
step-by-step instructions about how to perform tasks in the implementation plan, such as data 
cleaning, merging, computations, and classification. As such, detailed specifications about all 
tasks in the processing stage of the accountability implementation framework is especially vital. A 
barometer to help gauge the quality of the specifications is whether a staff member who may not 
have the background knowledge about the accountability system but has the prerequisite skill set 
can complete tasks by following the instructions in the specifications. As with the issues tracking 
log, ownership is a crucial element to the short and long-term utility of specifications. A person 
or group should be designated as responsible for maintaining, updating, and keeping everyone 
informed about the specifications. An annual kickoff meeting can be convened, during which all 
parties walk through key steps of the implementation process as described in the specifications. 
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Replication. |n the context of implementation, replication means assigning multiple people to 
independently carry out the steps described in the specifications and verifying that they yield the 
same results. Two independent internal replicators are usually sufficient for most implementation 
scenarios, especially if there are staffing or resource constraints, such as available software licenses. 
However, if a new and substantially different accountability system is being rolled out, or if the 
impact of the accountability results is far-reaching or of high profile, then a state should consider 
including additional replicators, potentially external to the SEA. One important consideration for 
replication is the definition or criteria for matching; that is, how similar do the results need to be to 
be considered “close enough.” In the ideal case, exact matching of results would be required and 
should be the preference. However, factors such as software or system settings and rounding rules 
could lead to occasional slight discrepancies in the results across replicators. Requiring replicators 
to have exact matching on all results often lead to sharing or copying of code or algorithms, 

which violates the goal of independence in the replication process. One approach of determining 
acceptable tolerance for discrepancies is to conduct an impact analysis — with simulated or real 
data — that “plays out” how the differences in results would ultimately affect key outcomes such 

as the classification and identification of schools. If no schools are negatively impacted, then the 
discrepancies can be considered acceptable. 


Test Cases. In engineering, a test case is a set of conditions or variables under which one can 
determine whether the (software or hardware) product, system, or process of interest is operating 
as intended. A collection of test cases, often referred to as a test suite, is usually defined as part of 
the design phase and is an integral part of the implementation plan. The test suite should include 
test cases that represent common or typical sets of condition as well as atypical, extreme or even 
out of bound conditions. The expected values or outcome for each test case should be specified 
in advance as part of the design. In the context of an accountability system, it is recommended 
that a state conduct empirical modeling, using either real or simulated data, of its accountability 
model to evaluate the reasonableness of its design and inform decision making. The results of 

the data modeling can also serve as the basis for test cases during implementation. For example, 
if the system is implemented correctly, one would expect a very high correlation for the school 
indicator scores and ratings and a very high degree of consistency for the school classifications 
and identification statuses between the modeled and operational data. A state can also define 
test cases based on well-known schools or districts to evaluate whether their accountability results 
are reasonable given their expected performance. The test suite should be well-documented and 
potentially integrated into the implementation process so that the test cases are automatically run 
each year. The test suite can also be updated or expanded as part of the continuous improvement 
process for the accountability system. 


Reasonableness Review. An often overlooked and potentially costly misconception assumed 

in assessment and accountability implementation is that if results from independent replicators 
match, then they must be correct. The fact is, even with the most carefully thought out, modeled, 
and tested system, scenarios that are unanticipated or unprecedented during the design 

phase can manifest in operational processing. Thus, beyond requiring an acceptable degree 


of matching across independent replications, the accountability results should also go through 

a reasonableness review. The main question asked in this review is “Do the accountability 

results make sense?” The reviewer takes a more macro view of the accountability results by 
considering the meaning and implications of the outcomes and looking for patterns or trends 
that are unusual. If irregularities or oddities are found, the reviewer attempts to find a reasonable 
explanation, which often requires examining additional information or supporting evidence. 
These may include outcomes from previous years or results disaggregated by grade level, 
content area, or student groups. 


The reasonableness review is usually a more difficult validation process than matching across 
replications or satisfying test cases because it tends to be more holistic and judgmental in 
nature. The outcome can, therefore, be reviewer-dependent and feel somewhat arbitrary. 
Developing guidelines or checklists for the reasonableness review can help make the process 
more concrete and standardized. The perspective an SEA should take with the reasonableness 
review is that someone else in the field — whether it is the media, accountability experts, district 
or school personnel, or other stakeholders — will likely be examining the accountability results 
through the same lens. It is also critical to document the reasonableness review to help establish 
evidence that the system is working as intended. Thus, conducting such a review helps the state 
build its case for the validity of the accountability system and anticipate potential feedback it 
may receive from the field. 
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CONCLUSION 


Having a sound implementation plan and sound quality control processes are instrumental in 
meeting the vision, priorities, and goals of the state’s accountability system. This brief gives 
guidance to states by providing a framework that highlights key components and considerations 
of a comprehensive implementation plan. It also suggests best practices for quality control to 

help ensure that the plan is executed with fidelity. Many states already have solid implementation 
plans and processes in place for their accountability systems‘. States are encouraged to share their 
resources and lessons learned with one another to leverage their experiences and to promote 

a culture of collaboration as each state enters the next phase of implementing its thoughtfully- 
designed accountability system. 


4 Agood example of an implementation timeline can be found here: http://sde.ok.gov/sde/sites/ok.gov.sde/ 
files/documents/files/Accountabilty%20Timeline_1.odf 
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